TOOLS ENGINEERED SPECIFICALLY FOR INFORMATION SPACE ORGANIZATION by STEPHEN

The burgeoning " information overload " problem plaguing today's technology users is well-documented. On nearly every front, the amount of data collected electronically far outstrips our ability to keep pace with understanding it. We have embedded devices with sensors that continually gather data; online digital libraries that regularly add new material as well as old; supermarket chains that capture and archive our every purchase; and of course the World Wide Web, which by conservative estimates is growing at a rate of over a hundred pages per minute[78]. It's no secret that by themselves, these vast megabytes mean nothing. In general, only the interpretation and deep analysis by human beings gives rise to productive knowledge that can be of practical benefit. Anything less is a waste of time. The tools currently available to assist in this quest for knowledge are principally two: query languages, in their various forms, and the discipline of data mining. The former is well-known to virtually any consumer of data – users (or programs) pose a series of concrete inquiries to some kind of interface, which attempts to extract bits of knowledge from an enormous data store. Sometimes a query is an intricate piece of code, such as an SQL statement, that navigates and pulls together a series of tables and values to arrive at a precise answer. In other cases, for instance a Google web search, it's more of a shot in the dark: a group of terms suggests the desired content which the search engine aims to find. In either case, between the human and the information lies a narrow channel through which questions are posed and results given. Researchers in the fields of query optimization and information retrieval have invested tremendous effort in improving this channel, in order to make the results speedier and more satisfactory. Data mining takes a different approach; the emphasis is less on ad hoc interaction and more on the discovery of patterns deeply embedded in the data which can only be uncovered by exhaustive statistical analysis. In some cases, human observers bring a great deal of structure to the data at the outset: they know the relevant categories to which individual data records belong, and seek complex rules by which future records can be automatically classified. Or they may approach a large data set with a particular subset of variables in mind, and want to verify mathematical correlations between …